Is Magnitude Estimation Worth the Trouble?

In sentence acceptability experiments, subjects are typically asked to indicate their response to sentences in one of three ways. In a yes/no forced choice task, subjects simply indicate whether or not the sentence sounds good. In an n-point numerical scale (or Likert scale) task, the extremes of a subjects choose a number on the scale that reflects their overall response to the sentence. Finally, in a magnitude estimation (ME) task, subjects compare experimental sentences to a reference sentence. This reference sentence is associated with a number (or subjects may choose this number on their own). Subjects are instructed to rate the experimental sentences in relation to the number given to the reference sentence. If the experimental sentence sounds twice as good as the reference sentence, for instan should divide it in half, etc. Each of these response methods has potential advantages and disadvantages. The yes/no method has the virtue of being very easy for subjects to understand, but it is often thought to be relatively coarse-grained and to require large numbers of subjects to detect fine differences. The n-point numerical scale arguably yields finer-grained results, while still being easy for subjects, but there is no guarantee that the chosen scale will allow as many distinctions in acceptability as subjects actually perceive or that subjects will treat the distance between any two adjacent points on the scale as being the same (e.g. in a 5-point scale, subjects might treat the difference between 1 and 2 as being larger or smaller than that between 3 and 4). ME is clearly not easy for subjects to understand, in that it is an unfamiliar task that requires some mathematical sophistication, but it could reasonably be expected to overcome the two disadvantages seen for the n-point scale. Subjects are able to make as many distinctions as they want, and since they are explicitly asked to make ratio judgments (i.e. how many times better or worse the experimental sentence is compared to the reference sentence), one would expect less uncertainty about the nature of the results. In this study, we submit these three response methods to a critical examination by comparing the results obtained by each in three otherwise identical experiments. In section 2, we review the previous literature on these methods, and in section 3, we present the set of experiments that constitute the core of our contribution, concluding that some of the claimed advantages of ME do not appear to be empirically supported. We devote some attention to differences among the three methods that emerged in our results in section 4 and we explore some other results of interest in our experiments in section 5. Section 6 presents conclusions and implications for the working syntactician.